Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nan to -inf #36

Merged
merged 1 commit into from
Nov 2, 2023
Merged

nan to -inf #36

merged 1 commit into from
Nov 2, 2023

Conversation

danjust
Copy link
Contributor

@danjust danjust commented Nov 2, 2023

Another tiny fix: Currently, diverging models that return nan get perfect evaluation (since candidate_score>torch.nan = False). The alternative to this fix would be to compute n_worse = torch.sum(candidate_score <= pos_score, dim=-1)

@danjust danjust requested a review from AlCatt91 November 2, 2023 13:30
AlCatt91
AlCatt91 approved these changes Nov 2, 2023
View reviewed changes
Copy link
Collaborator

@AlCatt91 AlCatt91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Daniel, good point! It might be slightly more efficient (?) computing the n_worse as you suggested, but I don't expect this nan_to_num_ replacement to add too much overhead to the metric computation, so I think it's also fine to leave it like this.

@danjust danjust merged commit 5d2ec26 into main Nov 2, 2023
1 check passed
@danjust danjust deleted the nan_eval branch November 2, 2023 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AlCatt91 AlCatt91 AlCatt91 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@danjust @AlCatt91